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associated with a distinct haptic logo and include their haptic logos in various messages sent to the handheld communication devices 

^ of their customers. In another embodiment, haptically-enabled avatars can be used in a chat session between users of handheld 



WO 2004/053644 



PCT7US2003/038868 



USING HAPTIC EFFECTS TO ENHANCE INFORMATION CONTENT IN 

COMMUNICATIONS 

INVENTORS 
Danny Grant, Jeffrey Eid, and Shoichi Endo 
CROSS-REFERENCE TO RELATED APPLICATIONS 

This application claims priority to U.S. Provisional Patent Application No. 
60/431,662, filed on December 8, 2002, the entire disclosure of which is incorporated herein 
by reference. 

FIELD OF THE INVENTION 

This invention relates generally to haptic-feedback systems. More specifically, 
embodiments of the present invention relate to using customized haptic effects in a variety of 
applications to convey information to users of handheld communication devices. 

BACKGROUND 

As handheld communication devices become part of everyday life, device 
manufactures and service providers strive to enhance the versatility and performance of such 
devices. 

Handheld communication devices in the art (e.g., mobile phones, pagers, personal 
digital assistants (PDAs), etc.) typically use auditory and visual cues to alert a user when 
incoming messages, such as voice calls and emails, are received. Such auditory and visual 
alerts, however, have the disadvantages of being distracting in some situations (e.g., during 
driving), or annoying in others (e.g., during a meeting or a concert). Although vibratory 
alerts are made available in some communication devices such as cellular phones, such 
vibratory effects cannot be customized or personalized according to applications, thus 
conveying little information to the user. A need, therefore, exists in the art for a new sensory 
modality that delivers information to users of handheld communication devices in a 
personalized fashion. 

SUMMARY 

Embodiments of the invention relate to methods and systems for providing 
-customized~haptic^essaging!l-to.-users-of-handheld-Communication devices~in_a. variety of 
applications. 

In one embodiment, a method of haptic messaging includes: receiving an input signal; 
extracting a haptic code from the input signal, the haptic code being associated with a haptic 
logo; and providing a control signal to an actuator, the control signal being based at least in 
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DETAILED DESCRIPTION 

Embodiments described in the following description are provided by way of example 
to illustrate some general principles of the invention, and should not be construed as limiting 
the scope of the invention in any manner. One skilled in the art would also recognize that 
various changes and modifications can be made herein, without departing from the principles 
and scope of the invention. 

FIG. 1 depicts a block diagram of a handheld communication device 100 according to 
an embodiment of the invention. It will be appreciated that various elements are shown in 
schematic form for illustrative purposes and are not drawn to scale. It will also be 
appreciated that many alternative ways of practicing the present invention exit. Accordingly, 
various changes and modifications may be made herein, without departing from the 
principles and scope of the invention. 

Device 100 includes a device body including a housing 110 and a user-interface 112; 
a processor 120; at least one actuator 130 in communication with processor 120; and a 
memory 140 in communication with processor 120. Device 100 also includes an antenna 150 
and a transceiver 160, in communication with processor 120. Device 100 additionally 
includes a display module 170 and an audio module 180, in communication with processor 
120. Display module 170 may include, for example, a liquid crystal device. Audio means 
180 may include, for example, a speaker, a microphone, and the like. 

For purpose of illustration in the embodiment of FIG. 1, processor 120, actuator 130, 
and memory 140 are shown to be enclosed within and coupled to the device body. Such an 
illustration, however, should not be construed as limiting the scope of the invention in any 
manner. In alternative embodiments, actuator 130 may, for example, be coupled to the 
outside of housing 110, or embedded in housing 110 via a suitable mechanism. Further, user- 
interface 112 may include one or more user-interface members. As used herein, a user- 
interface member includes, without limitation, a key pad having one or more keys, one or 
more buttons, a touch screen or touch pad, a scroll wheel, a direction pad, a trackball, a knob, 
a miniature joystick, or other user-interface means known in the art. 

Device 100 further includes an API (Application Program Interface) 190, working in 
conjunction -with-an-operating-system-1 95. A-deviee -driver— (not shown)-may optionally 
provide an interface between operating system 195 and processor 120. 

Memory 140 of device 100 stores a program code that includes instructions to cause 
processor 120 to perform various tasks. The following description provides some examples. 
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In another embodiment, the input signal may be associated with a reminder event, 
which may be a self-generated message on the handheld communication device serving as a 
reminder for a pre-scheduled activity (e.g., an appointment or a meeting). The source in this 
scenario may be associated with the type of a pre-scheduled activity (e.g., a business meeting 
vs. a restaurant reservation), or the time at which the pre-scheduled activity takes place. 

In yet another embodiment, the input signal may include a communication signal 
associated with a status event, for example, received via antenna 150 and transceiver 160 of 
FIG. 1. Examples of a status event include, but are not limited to: an advertisement (e.g., 
sale) event, a one-to-one marketing event, a business-transaction event, a stock-trading event, 
a weather- forecast event, a sports (or game) event, an entertainment event, and an emergency 
(e.g., 911) event. In this scenario, the source may be associated with a characteristic that 
distinctly identifies the sender and/or the nature of a status event, such as the phone number 
of the handheld user's stock broker, the e-mail address of the user's favorite store, the logo 
associated with the user's favorite TV or radio station, and so on. 

In one embodiment, an event of interest can be accompanied by a distinct haptic 
effect, or overlapping haptic effects, conveying to the user customized information such as 
"who is calling," "what is happening," and so on. The user can also be allowed to update the 
haptic lookup table, e.g., to include new events, and/or to modify the mappings between the 
existing events of interest and the corresponding haptic effects. 

Moreover, a specific haptic effect can be assigned to any incoming signal event whose 
source is unknown, so as to alert the user that the incoming message is from an un- 
identifiable or sender. 

As used herein, the term "handheld communication device" includes, without 
limitation, a mobile phone such as a cellular phone or a satellite phone, a personal digital 
assistant (PDA), a cordless telephone, a pager, a two-way radio, a handheld or portable 
computer, a game console controller, a personal gaming device, an MP3 player, or other 
personal electronic devices known in the art that are equipped with communication or 
networking capabilities. 

In one embodiment, the aforementioned haptic effects can be used as haptic ringers 
— feg.,--countei^ -convey 
information to the user about various events of interest. By way of example, a haptic ringer 
associated with a call from a loved one (e.g., the user's spouse) may comprise low-amplitude 
and high frequency vibrations that impart gentle sensations to the user. In contrast, a haptic 
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associated with the haptic logo. Step 330 may include providing a control signal to an 
actuator coupled to the handheld communication device, where the control signal is based at 
least in part on the haptic code and configured to cause the actuator to output the haptic 
effect. 

In one embodiment, the extracted haptic code may be directly applied to the actuator 
for rendering the desired haptic effect. In another embodiment, the haptic code may be 
configured according to a predetermined scheme or protocol that includes, for example, a 
table of haptic codes (some of which may be associated with one or more haptic logos) versus 
control signals for rendering the corresponding haptic effects. In this way, a processor in the 
handheld communication device can look up the corresponding control signal from the table 
based on the extracted haptic code, and output the selected control signal to the actuator for 
rendering the desired haptic effect. 

In the embodiments of FIG. 2 or 3, the handheld communication device (or the haptic 
code) may be programmed such that the haptic effect is output immediately, or at a 
prescribed time after receiving the input signal, as desired in applications. The haptic effects 
can also be triggered by, or synchronized with, other occurrences. 

A handheld communication device may be further configured such that some of its 
user-interface members (such as those described above) are each associated with a haptic 
code, e.g., according to a predetermined scheme or protocol. In one embodiment, some of 
these haptic codes may be associated with haptic effects that emulate expressions or 
behaviors, such as "laugh," "giggle," "hug," "high-five," "heartbeat," "pet purring," etc. This 
allows haptic effects to be transmitted and experienced, e.g., in an interactive conversation or 
a chat session, by pressing or manipulating such members. 

By way of example, suppose that user A (termed "Alice" herein) is engaged in a chat 
session with user B (termed "Bob" herein) via their respective mobile phones. In one 
embodiment, when Bob tells Alice a joke, Alice can respond by sending a "laugh" sensation 
to Bob, e.g., by pressing a key on her mobile phone that is assigned with a haptic code 
corresponding to a laugh sensation. This causes a signal to be transmitted from Alice's phone 
to Bob's phone, and a corresponding haptic effect to be output to Bob's phone (and thereby 
experienced by-Bob), — In-altemati ve-embodiments,-Alice-can-include^a haptic code in an 
outgoing message (which may also contain a video image such as a picture taken by her 
mobile phone, and/or a graphical feature such as an emoticon emulating a smiley face) to be 
transmitted to Bob, e.g., by pressing the corresponding user-interface member. The haptic 
code causes a haptic effect to be output when the message is delivered to a remote device 
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In another embodiment, the input signal of FIG. 5 may be associated with a "virtual 
touch," e.g., to mimic a handshake, a "high-five," a pat on the back, a pulse or heartbeat 
sensation, a pet purring sensation, or other touch sensations associated with human (and/or 
human-animal) interactions. In one scenario, the input signal at step 510 may include a 
"virtual touch indicator," based on which the request for a contact with a particular user- 
interface member is made. The virtual touch indicator may be in the form of a haptic code, a 
message, or other informative means. The control signal at step 530 may be generated, e.g., 
based on the virtual touch indicator, a haptic code associated with the user-interface member 
at play, or other predetermined scheme. The input signal at step 510 may also include a 
virtual touch indicator along with a virtual touch signal for rendering the desired haptic 
effect. In this case, the control signal at step 530 may be based on the virtual touch signal. 

Referring back to the chat session between Alice and Bob, by way of example at the 
end of their chat session, Alice may wish to send Bob a "high-five." She sends to Bob's 
mobile phone a signal including a virtual touch indicator, which in turn prompts a request that 
Bob be in contact with a user-interface member coupled to his phone, such as a direction pad 
(e.g., by putting his fingers on the individual keys of the direction pad), a key pad, a touch 
screen, a trackball, a joystick, or the like. The control signal for rendering a haptic effect that 
emulates a "high-five" may be based on the haptic code associated with the user-interface 
member, transmitted with the input signal from Alice, and/or other predetermined scheme. 

Interactive virtual touch can also be engaged between users of handheld 
communication devices, where the manipulation of a user-interface member on one handheld 
communication device is transmitted possibly in substantially real-time to another handheld 
device and experienced by its user, and vice versa. FIG. 6 depicts a flowchart 600 illustrating 
a method of providing interactive virtual touch in one embodiment of the present invention. 
In the embodiment shown, a handheld communication device first receives an input signal 
including a virtual touch indicator at step 610. A distinctive haptic ringer may, for example, 
accompany the arrival of the virtual touch indicator, identifying the sender and the nature of 
the input signal. The handheld communication device may then perform any necessary 
initialization to enable the communication at step 620, which may also include requesting a 
xontact-vvith^-particular user^nteriace-^ember~-coupJed-4o^h 

device at step 625. The handheld communication device subsequently receives a virtual 
touch signal in the communication associated with the desired haptic effect at step 630. The 
handheld communication device provides the haptic effect at step 640, e.g., by applying the 
virtual touch signal to an actuator coupled to the user-interface member. 
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output the haptic effect. Further, the input signal at step 810 may be received from GPS, a 
digital compass, or other navigation systems known in the art. 

In one embodiment, the haptic effect may be associated with a distance between the 
position of the handheld communication device and a predetermined location (termed 
5 "destination" herein). For example, the haptic effect may include a vibration having a 
magnitude and a frequency, where at least one of the magnitude and the frequency decreases 
as the distance from the destination diminishes. Additionally, the haptic effect may be 
configured to convey a quantitative measure of the distance. By way of example, the haptic 
effect may include one or more pulse or jolt sensations, where the number of pulses is 
10 proportional to the number of miles between the position of the handheld device and the 
destination. 

Processors described above (including processor 120 of FIG. 1) can include, for 
example, one or more digital logical processors capable of processing input, execute 
algorithms, and generate output as necessary to perform various tasks, such as those 

15 described above. Such processors/controllers may include a microprocessor, an Application 
Specific Integrated Circuit (ASIC), and state machines. Such processors include, or may be 
in communication with, media (including memory 140 of FIG. 1). Such media include, for 
example, computer readable media, which stores program code that, when executed by a 
processor, cause the processor to perform the steps described herein. Embodiments of 

10 computer-readable media include, but are not limited to, an electronic, optical, magnetic, or 
other storage or transmission device capable of providing a processor, such as the processor 
in a web server, with computer-readable instructions. Other examples of suitable media 
include, but are not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip, ROM, 
RAM, ASIC, configured processor, all optical media, all magnetic tape or other magnetic 

>5 media, or any other medium from which a computer processor can read. Also, various other 
forms of computer-readable media may transmit or carry instructions to a computer, 
including a router, private or public network, or other transmission device or channel. 

Program code and associated application programs related to various applications may 
also reside on a remote source, such as a network resource, a Web server, a remote handheld 

40 eonimuniGatioiv<leviGe^r-^mputer^^ handheld 

communication device on a regular or predetermined basis. Haptic effects (along with 
associated control signals) can also be downloaded or transmitted from a remote source, as 
described above. 



WO 2004/053644 PCT7US2003/038868 

i rvi Ail"* i 

configured to provide a third haptic effect that is different from the two mentioned above, 
e.g., when an e-mail is received from the e-mail address of Bob's stock broker (where the e- 
mail contains a "smiley-face" emoticon, for instance). The third haptic effect can be a 
vibration with high magnitude and short duration, e.g., to emulate a "high-five." 

In another embodiment, Bob can be watching a movie in a theater with his mobile 
phone in his pocket. It is set to make no noise, because Bob is in a theater. While Bob is 
watching the movie, Bob's mobile phone vibrates with the second haptic effect mentioned 
above. Bob chooses to ignore the call, because he does not wish to speak with his supervisor 
at a movie theater. Later, Bob's mobile phone vibrates with the first haptic effect. Bob wants 
to speak with Alice, for example, to make plans to meet later. So Bob answers the phone and 
quickly exits the theater to talk with Alice. 

Bob's mobile phone can also include a personal schedule/calendar application. After 
speaking with Alice, Bob can enter an entry in the calendar at the 7:00PM time mark - "Meet 
Alice". Bob can also choose a fourth haptic effect to associate with the calendar entry. The 
mobile phone can be programmed to output the fourth haptic effect fifteen minutes before the 
time entry of the calendar (i.e., at 6:45PM). 

Bob's mobile phone can be equipped with GPS capability, along with an associated 
application program for location determination. Bob can also store addresses of various 
locations of interest in the application program. In one embodiment, Bob can be on the road. 
Bob's mobile phone vibrates with a distinct fifth haptic effect. Bob recognizes the fifth 
haptic effect being associated with the haptic logo of his favorite electronics store. He then 
checks with the application program, and receives a sixth haptic effect associated with the 
distance between his current position and the store location. Bob then decides to make a stop 
at the store. 

A haptically-enabled handheld communication device of the invention may be further 
used as a two-way haptic remote control, for example, for controlling a remote system such 
as a Television set or a multimedia system. In one embodiment, the events as referred to 
above may be related to program channels shown on the remote system, each identified by a 
channel number (which may be used as the "source"), for instance. The corresponding haptic 
effeGts^ayJ^e^ustomized-oiv^^er-c^ inform a 

user as to which channel is on, as a user is channel-surfing by way of this haptic remote 
control, so that the user need not to look up the display screen. 

FIG. 9 depicts a flowchart illustrating a method for providing haptic effects to a 
remote control in one embodiment of the present invention. In the embodiment shown, the 
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That which is claimed is: 

1 . A method, comprising: 
receiving an input signal; 

extracting a haptic code from the input signal, the haptic code being associated with a 
5 haptic logo; and 

providing a control signal to an actuator, the control signal being based at least in part 
on the haptic code and configured to cause the actuator to output a haptic effect associated 
with the haptic logo. 

2. The method of claim 1 wherein the haptic logo is associated with a status event. 

10 3. The method of claim 2 wherein the status event includes one of an advertisement 
event, a business-transaction event, a one-to-one marketing event, a stock-trading event, a 
weather-forecast event, an entertainment event, a sports event, and an emergency event. 
4. The method of claim 1 wherein the haptic effect is output to a handheld 
communication device. 

15 5. A method, comprising: 

receiving an input signal associated with a chat message; 

causing an avatar associated with the chat message to be displayed on a handheld 
communication device; and 

outputting a control signal to an actuator coupled to the handheld communication 
10 device, the control signal configured to cause the actuator to output a haptic effect associated 
with the chat message. 

6. The method of claim 5 further comprising extracting a haptic code from the input 
signal, the control signal being based at least in part on the haptic code. 

7. The method of claim 5 further comprising correlating the haptic effect is with an 
!5 expression of the avatar. 

8. A computer-readable medium on which is encoded program code, comprising: 
program code for receiving an input signal; 

program code for extracting a haptic code from the input signal, the haptic code being 
associated with a haptic logo; and 

-ijQ program c od e fo r providing a contro l s igna l t o a n ac tu at or , the c ontr ol si g nal b ein g — 

based at least in part on the haptic code and configured to cause the actuator to output a 
haptic effect associated with the haptic logo. 

9. The computer-readable medium of claim 8 wherein the haptic logo is associated with 
a status event. 
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an actuator in communication with the processor; and 

a memory in communication with the processor, the memory storing program code 
executable by the processor, including: 

program code for receiving an input signal; 
5 program code for extracting a haptic code from the input signal, the haptic 

code being associated with a haptic logo; and 

program code for providing a control signal to the actuator, the control signal 
being based at least in part on the haptic code and configured to cause the actuator to 
output a haptic effect associated with the haptic logo. 
10 18. The apparatus of claim 17 wherein the actuator is coupled to a handheld 
communication device. 

19. The apparatus of claim 18 wherein the handheld communication device includes one 
of a cellular phone, a satellite phone, a cordless phone, a personal digital assistant, a pager, a 
two-way radio, a portable computer, a game console controller, a personal gaming device, 

15 and an MP3 player. 

20. The apparatus of claim 17 wherein the haptic logo is associated with a status event. 

21. The apparatus of claim 20 wherein the status event includes one of an advertisement 
event, a business-transaction event, a one-to-one marketing event, a stock-trading event, a 
weather-forecast event, an entertainment event, a sports event, and an emergency event. 

20 22. The apparatus of claim 17 wherein the memory further stores a haptic lookup table 
associating a plurality of haptic codes each with a control signal. 

23. The apparatus of claim 22 wherein the memory further stores program code to 
download the haptic lookup table from a remote source. 

24. The apparatus, comprising: 
>5 a processor, 

a display module in communication with the processor; 
an actuator in communication with the processor; and 

a memory in communication with the processor, the memory storing program code 
executable by the processor, including: 

SO ■ — : program code for receiving an input signal associated with a c h at m essag e^ 

program code for causing an avatar associated with the chat message to be 
displayed on the display module; and 
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